Bone and tool tracking in robotized computer-assisted surgery

ABSTRACT

A system for tracking at least one bone in robotized computer-assisted surgery, comprises a processing unit and a non-transitory computer-readable memory communicatively coupled to the processing unit and comprising computer-readable program instructions executable by the processing unit for: obtaining backscatter images of the at least one bone from a tracking device in a coordinate system; generating a three-dimensional geometry of a surface of the at least one bone from the backscatter images, the three-dimensional geometry of the surface being in the coordinate system; determining a position and orientation of the at least one bone in the coordinate system by matching the three-dimensional geometry of the surface of the at least one bone to a three-dimensional model of the bone; controlling an automated robotized variation of at least one of a position and orientation of the tracking device as a function of a processing of the backscatter images; and continuously outputting the position and orientation of the at least one bone in the coordinate system to a robot driver controlling a robot arm supporting a surgical tool in the coordinate system for altering the bone.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the priority of U.S. Patent ApplicationSer. No. 62/461,995, filed on Feb. 22, 2017, and the priority of U.S.Patent Application Ser. No. 62/529,745, filed on Jul. 7, 2017, both ofwhich are included herein by reference.

TECHNICAL FIELD

The present application relates to bone and tool tracking incomputer-assisted orthopedic surgery and in robotized computer-assistedsurgery.

BACKGROUND OF THE ART

Tracking of surgical instruments or tools is an integral part ofcomputer-assisted surgery (hereinafter “CAS”). The tools are tracked forposition and/or orientation in such a way that information pertaining tobodily parts is obtained. The information is then used in variousinterventions (e.g., orthopedic surgery, neurological surgery) withrespect to the body, such as bone alterations, implant positioning,incisions and the like during surgery.

The tracking technologies may use different technologies, such asmechanical, acoustical, magnetic, optical and radio frequency (RF)tracking. Depending on the technology used, different types of trackablemembers are fixed, permanently or temporarily, to the item that needs tobe tracked. For instance, during Total Knee Replacement (TKR) surgery,trackable members are fixed to the limbs and to the different surgicalinstruments, and these trackable members are tracked by the trackingsystem. The CAS system calculates position and orientation dataassociated with the tracking, and the information displayed by thecomputer is used by the surgeon to visualize the position of theinstrument(s) being manipulated with respect to the limbs, or innumerical values.

Optical tracking is commonly used in different forms. For example,passive retroreflective components are provided on tools and bones. Inorder to obtain values for position and/or orientation, the opticalelements must be in the line of sight of the optical sensor device. Asother examples, structured light tracking and laser rangefinder trackingare known optical tracking technologies. One common constraint withoptical tracking systems is the requirement for a line of sight betweenimage acquisition devices and the objects to track. Accordingly, surgeryemploying optical tracking may be imposed a given orientation as afunction of the required visibility between the optical sensor apparatusand the optical elements. If the line of sight is disrupted, orthopedictracking may be paused, as a possible consequence. In automated roboticsurgery, the interruption of optical tracking may result in the need forhuman intervention. There remains room for improvement.

SUMMARY

In accordance with a first embodiment of the present disclosure, thereis provided a system for tracking at least one bone in robotizedcomputer-assisted surgery, comprising: a processing unit; and anon-transitory computer-readable memory communicatively coupled to theprocessing unit and comprising computer-readable program instructionsexecutable by the processing unit for: obtaining backscatter images ofthe at least one bone from a tracking device in a coordinate system;generating a three-dimensional geometry of a surface of the at least onebone from the backscatter images, the three-dimensional geometry of thesurface being in the coordinate system; determining a position andorientation of the at least one bone in the coordinate system bymatching the three-dimensional geometry of the surface of the at leastone bone to a three-dimensional model of the bone; controlling anautomated robotized variation of at least one of a position andorientation of the tracking device as a function of a processing of thebackscatter images; and continuously outputting the position andorientation of the at least one bone in the coordinate system to a robotdriver controlling a robot arm supporting a surgical tool in thecoordinate system for altering the bone.

Further in accordance with the first embodiment, controlling theautomated robotized variation comprises for example identifying an imageratio of the at least one bone relative to environment in thebackscatter images and controlling the automated robotized variation toincrease the image ratio.

Still further in accordance with the first embodiment, controlling theautomated robotized variation comprises for example creating field ofview data indicative of the orientation of the at least one bonerelative to the position and orientation of the tracking device, andselecting the position and orientation of the tracking device as afunction of a desired point of view of the at least bone.

Still further in accordance with the first embodiment, selecting theposition and orientation of the tracking device as a function of adesired point of view of the at least bone includes for exampledetermining an upcoming location of the surgical tool on the robot armfrom a surgical flow of surgery planning.

Still further in accordance with the first embodiment, obtainingbackscatter images of the at least one bone from the tracking device inthe coordinate system comprises for example obtaining backscatter imagesof the surgical tool; and generating the three-dimensional geometry ofthe surface of the at least one bone from the backscatter imagescomprises for example generating a three-dimensional geometry of asurface of the surgical tool from the backscatter images in thecoordinate system.

Still further in accordance with the first embodiment, continuouslyoutputting the position and orientation of the at least one bone in thecoordinate system includes for example continuously outputting theposition and orientation of the surgical tool to the robot drivercontrolling the robot arm supporting the surgical tool in the coordinatesystem.

Still further in accordance with the first embodiment, the position andorientation of the surgical tool obtained from the backscatter images isverified for example with a position and orientation of the surgicaltool provided by the robot driver and outputting a discrepancy.

Still further in accordance with the first embodiment, determining theposition and orientation of the at least one bone in the coordinatesystem includes for example determining the position and orientation ofthe surgical tool in the coordinate system by matching thethree-dimensional geometry of the surface of the surgical tool to athree-dimensional model of the surgical tool.

Still further in accordance with the first embodiment, obtainingbackscatter images of the at least one bone from a tracking device in acoordinate system includes for example obtaining the backscatter imagesin a low-frequency capture mode when the surgical tool is distal to thebone, and in a high-frequency capture mode when the surgical tool isproximal to the bone.

Still further in accordance with the first embodiment, obtaining thebackscatter images in the low-frequency capture mode or thehigh-frequency capture mode includes for example determining an upcominglocation of the surgical tool on the robot arm from a surgical flow ofsurgery planning.

Still further in accordance with the first embodiment, obtaining thebackscatter images of the at least one bone includes for exampleobtaining the backscatter images from a point of view on a toolinterfacing with an anatomical part surrounding the bone or with thebone.

Still further in accordance with the first embodiment, controlling theautomated robotized variation of at least one of the position andorientation of the tracking device includes for example controlling arobotized tracker arm of the tracking device.

Still further in accordance with the first embodiment, generating thethree-dimensional geometry of the surface of the at least one bone fromthe backscatter images includes for example generating an alteredsurface, and wherein determining the position and orientation of the atleast one bone in the coordinate system includes determining theposition and orientation of the altered surface in the bone.

Still further in accordance with the first embodiment, the position andorientation of altered surface is verified for example with a positionand orientation of a planned altered surface from surgery planning andoutputting a discrepancy.

Still further in accordance with the first embodiment, an interferencefrom the backscatter images of the at least one bone is identified forexample and an indication of interference is output for example.

Still further in accordance with the first embodiment, obtaining thebackscatter images includes for example obtaining the backscatter imagesfrom visible structured light.

In accordance with a second embodiment of the present disclosure, thereis provided a system for tracking at least one bone in robotizedcomputer-assisted surgery, comprising: a tracking device including atleast a structured light source and a camera to generate and obtain thebackscatter images; and a computer-assisted surgery controller forobtaining backscatter images of the at least one bone from the trackingdevice in a coordinate system, generating a three-dimensional geometryof a surface of the at least one bone from the backscatter images, thethree-dimensional geometry of the surface being in the coordinatesystem, determining a position and orientation of the at least one bonein the coordinate system by matching the three-dimensional geometry ofthe surface of the at least one bone to a three-dimensional model of thebone, controlling an automated robotized variation of at least one of aposition and orientation of the tracking device as a function of aprocessing of the backscatter images, and continuously outputting theposition and orientation of the at least one bone in the coordinatesystem to a robot driver controlling a robot arm supporting a surgicaltool in the coordinate system for altering the bone.

Further in accordance with the second embodiment, the computer-assistedsurgery controller includes for example a position optimizer module foridentifying an image ratio of the at least one bone relative toenvironment in the backscatter images, for controlling the automatedrobotized variation to increase the image ratio.

Still further in accordance with the second embodiment, furthercomprising a field of view navigator module creates for example field ofview data indicative of the orientation of the at least one bonerelative to the position and orientation of the tracking device, andselects for example the position and orientation of the tracking deviceas a function of a desired point of view of the at least bone.

Still further in accordance with the second embodiment, the field ofview navigator module selects for example the position and orientationof the tracking device by determining an upcoming location of thesurgical tool on the robot arm from a surgical flow of surgery planning.

Still further in accordance with the second embodiment, obtainingbackscatter images of the at least one bone from the tracking device inthe coordinate system comprises for example obtaining backscatter imagesof the surgical tool, and generating the three-dimensional geometry ofthe surface of the at least one bone from the backscatter imagescomprises generating a three-dimensional geometry of a surface of thesurgical tool from the backscatter images in the coordinate system.

Still further in accordance with the second embodiment, continuouslyoutputting the position and orientation of the at least one bone in thecoordinate system includes for example continuously outputting theposition and orientation of the surgical tool to the robot drivercontrolling the robot arm supporting the surgical tool in the coordinatesystem.

Still further in accordance with the second embodiment, the position andorientation of the surgical tool obtained from the backscatter images isverified for example with a position and orientation of the surgicaltool provided by the robot driver and outputting a discrepancy.

Still further in accordance with the second embodiment, determining theposition and orientation of the at least one bone in the coordinatesystem includes for example determining the position and orientation ofthe surgical tool in the coordinate system by matching thethree-dimensional geometry of the surface of the surgical tool to athree-dimensional model of the surgical tool.

Still further in accordance with the second embodiment, obtainingbackscatter images of the at least one bone from the tracking device inthe coordinate system includes for example operating the tracking devicein a low-frequency capture mode when the surgical tool is distal to thebone, and in a high-frequency capture mode when the surgical tool isproximal to the bone.

Still further in accordance with the second embodiment, operating thetracking device in the low-frequency capture mode or the high-frequencycapture mode includes for example determining an upcoming location ofthe surgical tool on the robot arm from a surgical flow of surgeryplanning.

Still further in accordance with the second embodiment, the camera ofthe tracking device is located for example on a tool adapted tointerface with an anatomical part surrounding the bone or with the bone.

Still further in accordance with the second embodiment, generating thethree-dimensional geometry of the surface of the at least one bone fromthe backscatter images includes for example generating an alteredsurface, and wherein determining the position and orientation of the atleast one bone in the coordinate system includes for example determiningthe position and orientation of the altered surface in the bone.

Still further in accordance with the second embodiment, the position andorientation of altered surface is verified for example with a positionand orientation of a planned altered surface from surgery planning andoutputting a discrepancy.

Still further in accordance with the second embodiment, a robotizedtracker arm supports for example the tracking device, and wherein thecomputer-assisted surgery controller includes for example the robotdriver for controlling the robotized tracker arm, whereby controllingthe automated robotized variation of at least one of the position andorientation of the tracking device includes controlling the robotizedtracker arm of the tracking device.

Still further in accordance with the second embodiment, thecomputer-assisted surgery controller includes for example aninterference identifier module for identifying an interference from thebackscatter images of the at least one bone, whereby thecomputer-assisted surgery controller outputs an indication ofinterference.

Still further in accordance with the second embodiment, the structuredlight source produces for example structured light at least in a visiblelight spectrum.

In accordance with a third embodiment of the present disclosure, thereis provided a method for tracking at least one bone in computer-assistedsurgery, comprising: obtaining backscatter images of the at least onebone from a tracking device in a coordinate system, as positioned on atool interfacing with an anatomical part surrounding the bone or withthe bone; generating a three-dimensional geometry of a surface of the atleast one bone from the backscatter images, the three-dimensionalgeometry of the surface being in the coordinate system; determining aposition and orientation of the at least one bone in the coordinatesystem by matching the three-dimensional geometry of the surface of theat least one bone to a three-dimensional model of the bone; andcontinuously outputting the position and orientation of the at least onebone in the coordinate system to an interface for providing guidance inaltering the bone.

Further in accordance with the third embodiment, obtaining backscatterimages of the at least one bone from the tracking device in thecoordinate system comprises for example obtaining backscatter images ofthe surgical tool, and generating the three-dimensional geometry of thesurface of the at least one bone from the backscatter images comprisesfor example generating a three-dimensional geometry of a surface of thesurgical tool from the backscatter images in the coordinate system.

Still further in accordance with the third embodiment, continuouslyoutputting the position and orientation of the at least one bone in thecoordinate system includes for example continuously outputting theposition and orientation of the surgical tool in the coordinate system.

Still further in accordance with the third embodiment, determining theposition and orientation of the at least one bone in the coordinatesystem includes for example determining the position and orientation ofthe surgical tool in the coordinate system by matching thethree-dimensional geometry of the surface of the surgical tool to athree-dimensional model of the surgical tool.

Still further in accordance with the third embodiment, obtainingbackscatter images of the at least one bone from a tracking device in acoordinate system includes for example obtaining the backscatter imagesin a low-frequency capture mode when the surgical tool is distal to thebone, and in a high-frequency capture mode when the surgical tool isproximal to the bone.

Still further in accordance with the third embodiment, obtaining thebackscatter images in the low-frequency capture mode or thehigh-frequency capture mode includes for example determining an upcominglocation of the surgical tool from a surgical flow of surgery planning.

Still further in accordance with the third embodiment, generating thethree-dimensional geometry of the surface of the at least one bone fromthe backscatter images includes for example generating an alteredsurface, and wherein determining the position and orientation of the atleast one bone in the coordinate system includes for example determiningthe position and orientation of the altered surface in the bone.

Still further in accordance with the third embodiment, an automatedrobotized variation of at least one of a position and orientation of thetracking device is controlled for example as a function of a processingof the backscatter images, and the position and orientation of the atleast one bone in the coordinate system is continuously output forexample to a robot driver controlling a robot arm supporting a surgicaltool in the coordinate system for altering the bone.

Still further in accordance with the third embodiment, controlling theautomated robotized variation comprises for example identifying an imageratio of the at least one bone relative to environment in thebackscatter images and controlling the automated robotized variation toincrease the image ratio.

Still further in accordance with the third embodiment, controlling theautomated robotized variation comprises for example creating field ofview data indicative of the orientation of the at least one bonerelative to the position and orientation of the tracking device, andselecting the position and orientation of the tracking device as afunction of a desired point of view of the at least bone.

Still further in accordance with the third embodiment, selecting theposition and orientation of the tracking device as a function of adesired point of view of the at least bone includes for exampledetermining an upcoming location of the surgical tool on the robot armfrom a surgical flow of surgery planning.

Still further in accordance with the third embodiment, the position andorientation of the surgical tool obtained from the backscatter images isverified for example with a position and orientation of the surgicaltool provided by the robot driver and outputting a discrepancy.

Still further in accordance with the third embodiment, controlling theautomated robotized variation of at least one of the position andorientation of the tracking device includes for example controlling arobotized tracker arm of the tracking device.

Still further in accordance with the third embodiment, obtaining thebackscatter images includes for example obtaining the backscatter imagesfrom visible structured light.

In accordance with a fourth embodiment of the present disclosure, thereis provided a system for tracking at least one bone in robotizedcomputer-assisted surgery, comprising: a processing unit; and anon-transitory computer-readable memory communicatively coupled to theprocessing unit and comprising computer-readable program instructionsexecutable by the processing unit for: obtaining images of the at leastone bone from a tracking device in a coordinate system, with at leastone patch on the bone, the patch having a trackable pattern thereon;associating a three-dimensional geometry of a surface of the at leastone bone to the patch, the three-dimensional geometry of the surfacebeing in the coordinate system; determining a position and orientationof the at least one bone in the coordinate system by matching thethree-dimensional geometry of the surface of the at least one bone to athree-dimensional model of the bone; controlling an automated robotizedvariation of at least one of a position and orientation of the trackingdevice as a function of a processing of the images; and continuouslyoutputting the position and orientation of the at least one bone in thecoordinate system to a robot driver controlling a robot arm supporting asurgical tool in the coordinate system for altering the bone.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of an automated robotic computer-assistedsurgery (CAS) system in accordance with the present disclosure;

FIG. 2 is a block diagram of a CAS controller and tracking controllerwith the automated robotic CAS system of FIG. 1;

FIG. 3 is a flow diagram of a method for tracking objects in robotizedcomputer-assisted surgery; and

FIG. 4 is a perspective view of a tracking camera on a retractor inaccordance with the present disclosure.

DETAILED DESCRIPTION

Referring to the drawings and more particularly to FIG. 1, an automatedrobotic computer-assisted surgery (CAS) system is generally shown at 10,and is used to perform orthopedic surgery maneuvers on a patient,including pre-operative analysis of range of motion and implantassessment planning, as described hereinafter. The system 10 is shownrelative to a patient's knee joint in supine decubitus, but only as anexample. The system 10 could be used for other body parts, includingnon-exhaustively hip joint, spine, and shoulder bones.

The CAS system 10 is robotized, and has or may have a robot arm 20, atracker arm 30, a tracking device 40, a CAS controller 50, a trackingcontroller 60 (FIG. 2), and a secondary tracking device 70:

-   -   The robot arm 20 is the working end of the system 10, and is        used to perform bone alterations as planned by an operator        and/or the CAS controller 50 and as controlled by the CAS        controller 50. The robot arm 20 may also be configured for        collaborative/cooperative mode in which the operator may        manipulate the robot arm 20. For example, the tooling end, also        known as end effector, may be manipulated by the operator;    -   The tracker arm 30 supports the tracking device 40. The tracker        arm 30 is robotized in that its movements can be controlled by        the CAS controller 50;    -   The tracking device 40 performs image acquisition in optical        tracking, using for instance structured light, or        three-dimensional (3D) camera tracking, also known as range        imaging, depth imaging, in contrast to structured light tracking        with structured light pattern projection;    -   The CAS controller 50 controls the robot arm 20 and the tracker        arm 30, and operates the tracking device 40. Moreover, as        described hereinafter, the CAS controller 50 may also drive the        robot arm 20 through a planned surgical procedure;    -   The tracking controller 60 is tasked with determining the        position and/or orientation of the various relevant objects        during the surgery procedure, such as the bone(s) and tool(s),        using data acquired by the tracking device 40. The position        and/or orientation is used by the CAS controller 50 to control        the robot arm 20 and the tracker arm 30.    -   The secondary tracking device 70 may optionally be used to track        the bones of the patient, and the robot arm 20. For example, the        secondary tracking device 70 may assist in performing the        calibration of the patient bone with respect to the robot arm        20, i.e. determining its position and orientation, for        subsequent navigation in a coordinate system (also known as        frame of reference, global reference system, etc).

Referring to FIG. 1, a schematic example of the robot arm 20 and of thetracker arm 30 is provided. The robot arm 20 may stand from a base 21,for instance in a fixed relation relative to the operating-room (OR)table supporting the patient, whether it is attached or detached fromthe table. The relative positioning of the robot arm 20 relative to thepatient is a determinative factor in the precision of the surgicalprocedure, whereby a foot support A1 and thigh support A2 may assist inkeeping the operated limb fixed in the illustrated X, Y, Z coordinatesystem. Although not shown, the foot support A1 and/or the thigh supportA2 could be automated to robotize the displacement and positioning ofthe patient's leg, and optionally to perform tests on the leg. The robotarm 20 has a plurality of joints 22 and links 23, of any appropriateform, to support a tool head 24 that interfaces with the patient. Forexample, the end effector or tool head 24 may optionally incorporate aforce/torque sensor for collaborative/cooperative control mode, in whichan operator manipulates the robot arm 20. The robot arm 20 is shownbeing a serial mechanism, arranged for the tool head 24 to bedisplaceable in a desired number of degrees of freedom (DOF). Forexample, the robot arm 20 controls 6-DOF movements of the tool head 24,i.e., X, Y, Z in the coordinate system, and pitch, roll and yaw. Feweror additional DOFs may be present. For simplicity, only a genericillustration of the joints 22 and links 23 is provided, but more jointsof different types may be present to move the tool head 24 in the mannerdescribed above. The joints 22 are powered for the robot arm 20 to moveas controlled by the CAS controller 50 in the six DOFs, and in such away that the position and orientation of the tool head 24 in thecoordinate system may be known, for instance by readings from encoderson the various joints 22. Therefore, the powering of the joints 22 issuch that the tool head 24 of the robot arm 20 may execute precisemovements, such as moving along a single direction in one translationDOF, or being restricted to moving along a plane, among possibilities.Such robot arms 20 are known, for instance as described in U.S. patentapplication Ser. No. 11/610,728, and incorporated herein by reference.The position and orientation of the tool head 24 may be calculated usingsolely the encoders on the various joints. The tracking device 40 and/orsecondary tracking device 70 may also be used for this purpose, or othersystems such as inertial sensor systems. It may be necessary to have thebase 21 of the robot arm 20 fixed in a known location relative to thetracker arm 30 or alternatively be marked with a tracking patterncompatible with the secondary tracking device 70.

In FIG. 1, the tool head 24 supports a burr 26A, used to resurface ordrill a bone. The tool head 24 may also comprise a chuck or like toolinterface, typically actuatable in rotation. The tool head 24 may havelaminar spreader plates, actuatable independently from a remainder ofthe tool head 24, for simultaneous use with a tool support by the toolhead 24. The laminar spreader plates are used to spread soft tissueapart to expose the operation site. The laminar spreader plates may alsobe used as pincers, to grasp objects, etc. As a non-exhaustive example,other tools that may be supported by the tool head 24 include aregistration pointer, a reamer (e.g., cylindrical, tapered), areciprocating saw, a retractor, a laser rangefinder or light-emittingdevice (e.g., the indicator device of U.S. Pat. No. 8,882,777) dependingon the nature of the surgery. The various tools may be part of amulti-mandible configuration or may be interchangeable, whether withhuman assistance, or as an automated process. The installation of a toolin the tool head 24 may then require some calibration in order to trackthe installed tool in the X, Y, Z coordinate system of the robot arm 20.

The tracker arm 30 may also stand from a base 31, for instance in afixed relation relative to the operating-room (OR) table supporting thepatient of the base 21 of the robot arm 20. The relative positioning ofthe tracker arm 30 relative to the patient is adjustable as describedbelow to ensure that the resected bone portion is tracked in theillustrated X, Y, Z coordinate system, relative to the tool head 24. Thetracker arm 30 has a plurality of joints 32 and links 33, of anyappropriate form, to support the tracking device 40. The tracker arm 30is shown being a serial mechanism, arranged for the tracking device 40to be displaceable in a desired number of degrees of freedom (DOF). Forexample, the tracker arm 30 controls 6-DOF movements of the trackingdevice 40, i.e., X, Y, Z in the coordinate system, and pitch, roll andyaw. Fewer or additional DOFs may be present. For simplicity, only ageneric illustration of the joints 32 and links 33 is provided, but morejoints of different types may be present to move the tracking device 40in the manner described above. The joints 32 are powered for the trackerarm 30 to move as controlled by the CAS controller 50 in the six DOFs,and in such a way that the position and orientation of the trackingdevice 40 may be known, for instance by readings from encoders on thevarious joints 32. Therefore, the powering of the joints 32 is such thatthe tracking device 40 at the end of the tracker arm 30 may executeprecise movements. The tracker arm 30 may be similar to robot arms asdescribed in U.S. patent application Ser. No. 11/610,728.

The tracking device 40 is mounted to the end of the tracker arm 30. Thetracking device 40 is of the type using light backscatter (a.k.a.backscattered radiation) to surgically track objects. In the presentdisclosure, the tracking device 40 may be used to track tools and bonesso as to guide the robot arm 20 in altering the bone based on surgeryplanning. Backscattered radiation can be used for acquisition of 3Dsurface geometries of bones and tools.

The tracking device 40 may produce structured light illumination fortracking objects with structured light 3D imaging. In structured lightillumination, a portion of the objects is illuminated with one ormultiple patterns from a pattern projector 41. Structured light 3Dimaging is based on the fact that a projection of a line of light fromthe pattern projector 41 onto a 3D shaped surface produces a line ofillumination that appears distorted as viewed from perspectives otherthan that of the pattern projector 41. Accordingly, imaging such adistorted line of illumination allows a geometric reconstruction of the3D shaped surface. Imaging of the distorted line of illumination isgenerally performed using one or more cameras 42 which are spaced apartfrom the pattern projector 41 so as to provide such differentperspectives, e.g., triangulation perspective. In some embodiments, thepattern projector 41 is configured to project a structured light gridpattern including many lines at once as this allows the simultaneousacquisition of a multitude of samples on an increased area. In theseembodiments, it may be convenient to use a pattern of parallel lines.However, other variants of structured light projection can be used insome other embodiments.

The structured light grid pattern can be projected onto the surface totrack using the pattern projector 41. In some embodiments, thestructured light grid pattern can be produced by incoherent lightprojection, e.g., using a digital video projector, wherein the patternsare typically generated by propagating light through a digital lightmodulator. Examples of digital light projection technologies includetransmissive liquid crystal, reflective liquid crystal on silicon (LCOS)and digital light processing (DLP) modulators. In these embodiments, theresolution of the structured light grid pattern can be limited by thesize of the emitting pixels of the digital projector. Moreover, patternsgenerated by such digital display projectors may have smalldiscontinuities due to the pixel boundaries in the projector. However,these discontinuities are generally sufficiently small that they areinsignificant in the presence of a slight defocus. In some otherembodiments, the structured light grid pattern can be produced by laserinterference. For instance, in such embodiments, two or more laser beamscan be interfered with one another to produce the structured light gridpattern wherein different pattern sizes can be obtained by changing therelative angle between the laser beams.

The pattern projector 41 may emit light that is inside or outside thevisible region of the electromagnetic spectrum. For instance, in someembodiments, the emitted light can be in the ultraviolet region and/orthe infrared region of the electromagnetic spectrum such as to beimperceptible to the eyes of the medical personnel. In theseembodiments, however, the medical personnel may be required to wearprotective glasses to protect their eyes from such invisible radiations.As alternatives to structured light, the tracking device 40 may alsooperate with laser rangefinder technology or triangulation, as a fewexamples among others.

The tracking device 40 further includes cameras 42 to acquirebackscatter images of the illuminated portion of objects. Hence, thecameras 42 capture the pattern projected onto the portions of theobject. The cameras 42 are adapted to detect radiations in a region ofthe electromagnetic spectrum that corresponds to that of the patternsgenerated by the light projector 41. As described hereinafter, the knownlight pattern characteristics and known orientation of the patternprojector 41 relative to the cameras 42, are used by the trackingcontroller 60 to generate a 3D geometry of the illuminated portions,using the backscatter images captured by the camera(s) 42. Although asingle camera spaced form the pattern projector 41 can be used, usingmore than one camera may increase the field of view and increase surfacecoverage. For instance, in the illustrated embodiment, a pair of cameras42 is used.

The tracking device 40 may also have one or more filters 46 (hereinafter“the filter 46”) integrated into either or both of the cameras 42 tofilter out predetermined regions or spectral bands of theelectromagnetic spectrum.

The filter 46 can be removably or fixedly mounted in front of any givencamera 42. For example, the filter 46 can be slidably movable into andout of the optical path of the cameras 42, manually or in an automatedfashion (e.g., using a motor or a solenoid). In some other embodiments,multiple filters may be periodically positioned in front of a givencamera in order to acquire spectrally resolved images with differentspectral ranges at different moments in time, thereby providing timedependent spectral multiplexing. Such an embodiment may be achieved, forexample, by positioning the multiple filters in a filter wheel that iscontrollably rotated to bring each filter in the filter wheel into theoptical path of the given one of the camera 42 in a sequential manner.

In some embodiments, the filter 46 can allow transmittance of only somepredetermined spectral features of objects within the field of view,captured either simultaneously by the tracking device 40 or separatelyby the secondary tracking device 70, so as to serve as additionalfeatures that can be extracted to improve accuracy and speed ofregistration.

More specifically, the filter 46 can be used to provide a maximumcontrast between different materials which can improve the imagingprocess and more specifically the soft tissue identification process.For example, in some embodiments, the filter 46 can be used to filterout bands that are common to backscattered radiation from typical softtissue items, the surgical structure of interest, and the surgicaltool(s) such that backscattered radiation of high contrast between softtissue items, surgical structure and surgical tools can be acquired.Additionally, or alternatively, where white light illumination is used,the filter 46 can includes band pass filters configured to let pass onlysome spectral bands of interest. For instance, the filter 46 can beconfigured to let pass spectral bands associated with backscattering orreflection caused by the bones, the soft tissue and/or markers 71A-Dwhile filtering out spectral bands associated with specifically coloreditems such as tools, gloves and the like within the surgical field ofview. Other methods for achieving spectrally selective detection,including employing spectrally narrow emitters, spectrally filtering abroadband emitter, and/or spectrally filtering a broadband imagingdetector (e.g., the camera 12), can also be used. Another light source48 may also be provided on the tracking device 40, for a secondarytracking option, as detailed below. It is considered to applydistinctive coatings on the parts to be tracked, such as the bone andthe tool, to increase their contrast relative to the surrounding softtissue.

In accordance with another embodiment, the tracking device 40 mayinclude a 3D camera to perform range imaging, and hence determineposition data from the captured images during tracking. The expression3D camera is used to describe the camera's capability of providing rangedata for the objects in the image it captures, but the 3D camera may ormay not produce 3D renderings of the objects it captures. In contrast tostructured light 3D imaging, range tracking does not seek specificillumination patterns in distance calculations, but relies instead onthe images themselves and the 3D camera's capacity to determine thedistance of points of objects in the images. Stated differently, the 3Dcamera for ranging performs non-structured light ranging, and theexpression “ranging” is used herein to designate such non-structuredlight ranging. Such range tracking requires that the 3D camera becalibrated to achieve suitable precision and accuracy of tracking. Inorder to be calibrated, the tracking device 40 may use a known visualpattern in a calibration performed in situ, at the start of thetracking, and optionally updated punctually or continuously throughoutthe tracking. The calibration is necessary to update the cameraacquisition parameters due to possible lens distortion (e.g., radial,rotational distortion), and hence to rectify image distortion to ensurethe range accuracy.

Therefore, the tracking device 40 with ranging camera may have a similarconfiguration as that of FIG. 1, although it may also be smaller, asshown in FIG. 4, in which the tracking device is a 3D camera 42 (asopposed to structured light 3D imaging system). Alternatively, thetracking device 40 may be provided with other equipment, for example forendoscopic tracking. The tracking device 40 may thus be equipped withapplication-specific lenses, such as a borescopic lens. In the trackingdevice 40 with ranging camera, component 41 may be a light projector orlight source to project light on the target area, if necessary. Forexample, the tracking device 40 used in endoscopic uses may require itslight source. Moreover, the light source may emit light within a givenwavelength, including within a non-visible range, such as infrared. Thetracking device 40 may feature one or more 3D cameras 42. Filters 46 mayalso be used to provide a maximum contrast between different materialswhich can improve the imaging process and more specifically the softtissue identification process, in a manner similar than as describedabove.

Referring to FIG. 2, the CAS controller 50 is shown in greater detailrelative to the other components of the robotized CAS system 10. The CAScontroller 50 has a processor unit to control movement of the robot arm20, and of the tracker arm 30. The robotized surgery controller 50provides computer-assisted surgery guidance to an operator through anautomated alteration or resection of bones, for instance in the form ofsurgical data updated during the surgical procedure. The CAS system 10may comprise various types of interfaces D, for the information to beprovided to the operator. The interfaces D may be monitors and/orscreens including wireless portable devices (e.g., phones, tablets),audio guidance, LED displays, among many other possibilities. Forexample, the interface D comprises a graphic user interface (GUI)operated by the system 10. The interface D may also display imagescaptured by the cameras 40 and/or 70, for instance to be used in thecollaborative/cooperative control mode of the system 10, or for visualsupervision by the operator of the system 10, with augmented reality forexample. The CAS controller 50 may drive the robot arm 20 in performingthe surgical procedure based on the surgery planning achievedpre-operatively, and controls the tracker arm 30 in suitably orientingand positioning the tracking device 40 to continuously track the tool 24relative to the anatomical features such as the bones. The CAScontroller 50 runs various modules, in the form of algorithms, code,non-transient executable instructions, etc, in order to operate the CASsystem 10 in the manner described herein. The CAS controller 50 may bepart of any suitable processor unit, such as a personal computer orcomputers including laptops and desktops, tablets, server, etc.

The controller 50 may hence have a robot driver module 51. The robotdriver module 51 is tasked with powering or controlling the variousjoints of the robot arm 20 and of the tracker arm 30 based on operatordemands or on surgery planning. As shown with bi-directional arrows inFIG. 2, there may be some force feedback provided by the robot arm 20and tracker arm 30 to avoid damaging the bones, and to detect contactbetween tool head 24, tracked device 40, and anatomical features.

The robot driver module 51 may perform actions based on a surgeryplanning 52. The surgery planning 52 may be a module programmedspecifically for any given patient, according to the parameters ofsurgery desired by an operator such as an engineer and/or surgeon. Theparameters may include geometry of selected, planned bone cuts, plannedcut depths, sequence or workflow of alterations with a sequence ofsurgical steps and tools, tools used, etc.

The tracking controller 60 may be a subpart of the CAS controller 50, oran independent module or system. The tracking controller 60 receivesfrom the tracking device 40 the backscatter images of the objects. Thetracking controller 60 processes the backscatter images to determine therelative position of the objects, and segment the objects from thebackscatter images. Accordingly, the tracking processor 60 is providedwith models of the objects to be tracked. For example, the trackingcontroller 60 may track bones and tools, and hence uses virtual bonemodels B and tool models C. The bone models B may be acquired frompre-operative imaging (e.g., MRI, CT-scans), for example in 3D or inmultiple 2D views, including with 2D X-ray to 3D bone modeltechnologies. The virtual bone models B may also include some imageprocessing done preoperatively, for example to remove soft tissue orrefine the surfaces that will be exposed and tracked. The virtual bonemodels B may be of greater resolution at the parts of the bone that willbe tracked during surgery, such as the knee articulation in kneesurgery. The bone models B may also carry additional orientation data,such as various axes (e.g., longitudinal axis, mechanical axis, etc).The bone models B may therefore be patient specific. It is alsoconsidered to obtain bone models from a bone model library, with thedata obtained from the backscatter images used to match a generated 3Dsurface of the bone with a bone from the bone atlas. The virtual toolmodels C may be provided by the tool manufacturer, or may also begenerated in any appropriate way so as to be a virtual 3D representationof the tool(s), such as the tool head 24. Additional data may also beavailable, such as tool orientation (e.g., axis data and geometry). Itis considered to provide specific detectable landmarks on the tool(s) toensure the detectable landmarks will be properly imaged and detected bythe tracking controller 60. In matching the 3D geometry to the bonemodels B, the tracking calculator 61 may reduce its computation usingdifferent strategies. According to one possibility, the surgicalplanning 52 may provide some guidance as to parts of the bones that arealtered during the surgical procedure. Likewise, the bone model(s) B mayhave higher resolution for the parts of the bone that will be alteredduring surgery. The remainder of the bone may be limited to informationon landmarks, such as axis orientation, center of rotation, midpoints,etc. A similar approach may be taken for the tool models C, with thefocus and higher detail resolution being on parts of the tools that comeinto contact with the bone, such as the tool head 24.

In an embodiment with structured light projection, the trackingcontroller 60 receives the backscatter images from the camera(s) 42, asa result of the structured light projection from the projector 41. Inanother embodiment, the tracking controller 60 receives the images fromthe ranging camera 42, and ensures that the ranging camera 42 iscalibrated for ranging to be done from the acquired images. An initialcalibration may be done using calibration pattern E. The calibrationpattern E is placed in the light of sight of the camera 42 such that itis imaged by the ranging camera 42. The calibration pattern E is anyappropriate shape and configuration, but may be a planar recognizablepattern with high contrast. The tracking controller 60 has a trackingcalculator module 61 that stores a virtual version of the calibrationpattern E, including precise geometrical data of the calibration patternE. The tracking calculator module 61 therefore performs a correspondencebetween imaged and virtual calibration patterns E. The correspondencemay entail calculating the mapping function between landmarks on theplanar imaged calibration pattern E and the virtual calibration patternE. This may include a projection of the calibration patterns E on oneanother to determine the distortion characteristics of the images of theranging camera 42, until the rectification values are determined by thetracking calculator module 61 to correct the images of ranging camera42. This calibration may be repeated punctually through the procedure,for instance based on the camera updating requirements. It may requirethat the camera 42 is used in conjunction with a calibration reflectivesurface whose position and orientation relative to the camera 42 isknown. The calibration may be automatically performed by the CAS system10.

The tracking controller 60 may therefore generate a 3D geometry from thebackscatter images, using the known patterns of structured light, orcalibrated camera images, along with the known shape of the virtual bonemodel(s) B and/or tool model(s) C. Moreover, the generated 3D geometrymay be located in the X, Y, Z, coordinate system using the knownpositional relation between the pattern projector 41 and the camera(s)42, in the case of structured light tracking, or the location of thecamera 42 in ranging. Therefore, as a first step, the trackingcalculator module 61 of the tracking controller 60 generates a 3Dgeometry of the portions of the object being illuminated. Then, usingthe virtual models B and/or C of the bone(s) and tool(s), respectively,the tracking controller 60 can match the 3D geometry with the virtualmodels B and C, with the objects detected being segmented. Consequently,the tracking controller 60 determines a spatial relationship between theobjects being illuminated and the preoperative 3D models, to provide adynamic (e.g. real time or quasi real time) intraoperative tracking ofthe bones relative to the tools. In an embodiment, the trackingcalculator module 61 only determines the position and orientation of thebone in the coordinate system, and locates the tool using other methods,such as obtaining the position and orientation of the tool from therobot driver 51 using the encoders in the robot arm 20. In anembodiment, the position and orientation of the surgical tool 24calculated by the tracking controller 60 may be redundant over thetracking data provided by the robot driver 51 and robot arm sensors.However, the redundancy may assist in ensuring the accuracy of thetracking of the surgical tool. For example, the redundancy is used as asafeguard against incorrect tracking from the CAS controller 50, forinstance due to bone movement or relative movement between the robot arm20 and the patient and/or table. The redundancy may also allow thereduction of frequency of image processing for the surgical tool 24.Also, the tracking of the tool 24 using the tracking device 40 may beused to detect any discrepancy between a calculated position andorientation of the surgical tool 24 through the sensors on the robot arm20, and the actual position and orientation of the surgical tool 24. Forexample, an improper mount of the tool 24 into the chuck of the robotarm 20 could be detected from the output of the tracking device 40, whenverified with the position and orientation from the robot driver 51(e.g., obtained from the encoders on the robot arm 20 or from thesecondary tracking device 70). The operator may be prompted to verifythe mount, via the interface D.

The tracking controller 60 may also use tracking patterns F to furtherassist in the tracking of tools and bones, in the case of rangingcameras. The tracking patterns F may or may not have reflectiveproperties, and their tracking may operate with backscatter. Moreparticularly, as shown concurrently in FIGS. 2 and 4, the trackingpatterns F may be on patches, plates, chips, affixed to the objects tobe tracked. In an embodiment, tracking patterns F may be affixed tomultiple bones of an articulation of the patient, such as the tibia andthe femur. Once the spatial relationship between images and 3D models ofbones and tools has been established for dynamic intraoperative trackingof the bones relative to the tools, the tracking device 40 and thetracking controller 60 may rely on the optical images of the trackingpatterns F for the dynamic intraoperative tracking. The position andorientation of the tracking patterns F relative to their respectivebones/tools is recorded as part of the spatial relationship by thetracking calculator module 61. As the tracking patterns F arecontrasting recognizable patterns, they may be more easily definable asobserved by the camera 42 than low contrast uniform items, such as abone surface. The tracking patterns F could be made from or coated withnon-stick material to prevent blood, bodily fluids or particulate matterfrom obscuring the pattern. Therefore, using optical detection, thetracking calculator module 61 uses the spatial relationship between thetracking patterns F and respective bone/tool for subsequent tracking.The spatial relationship may be established in situ, or may bepreprogrammed as well, especially for tools configured to receive thetracking patterns F in a predetermined way. The tracking patterns Fcould be made from bio-resorbable material such that they do not need tobe removed prior to closing the incision. The tracking patterns F mayeach have a unique pattern that has a single orientation (e.g., no axisof symmetry), so as to have their orientation trackable. It iscontemplated as well to have tracking patterns F with the same pattern,for example with steps performed to associate bone(s) and/or tool(s) toeach tracking pattern F. In an embodiment, the tracking patterns F areused in complementary fashion to the bone/tool structured light trackingdescribed above, for example as taking over for the structured lighttracking, or to validate the structured light tracking. In anotherembodiment, the tracking patterns F are used with the position data fromthe robot arm 20 for the tracking. The tracking patterns F may be in theform of a high resolution dark pattern on a light background, or viceversa, similar to a QR code, to a bar code, etc. In an embodiment, thepatterns F are on a flat surface, and thus are two-dimensional (2D). Thetracking patterns F may be less than 1.0 inch in diameter, or less than1.0 inch in width and/or in height. The tracking patterns F may beadhered, tacked, nailed, etc to the bone and/or to the tool.

The tracking device 40 may continuously capture backscatter images, forthe tracking controller 60 to perform a continuous tracking of theobjects. The frequency of capture may vary according to differentfactors. For example, there may be different phases during the surgicalworkflow, some in which the tracking requires a more dynamic update, andsome in which tracking updates are less important. Another factor thatmay affect the image capture frequency is the fixed relation of theobjects. For example, once the tracking controller 60 identifies a bonefrom the backscatter images, the frequency capture by the trackingdevice 40 may be reduced if the bone is fixed (e.g., by the foot supportA1 or tight support A2 of FIG. 1), if the bone alterations have not yetbegun. Also, when both a tool head 24 and a bone are tracked, thefrequency capture may be reduced when the tool head 24 and the bone arespaced from one another by a given distance, and increased as theproximity between the tool head 24 and the bone is increased. Thetracking calculator module 61 may drive the tracking device 40 in orderto control the frequency. For example, the tracking calculator module 61may adapt the frequency using the surgical planning 51, e.g.,anticipating upcoming steps in the workflow, etc. The trackingcalculator module 61 may consequently toggle between a low-frequencycapture mode and a high-frequency capture mode, for example. Thelow-frequency capture mode may be in instances in which the tool head 24is at a given distance from the bone, and is not driven to alter thebone. The low-frequency capture mode may also be operated when theobjects are in a fixed relation relative to one another. Other modes arecontemplated.

The tracking device 40 is on the tracker arm 30 for its position andorientation to be adjusted to ensure it provides suitable backscatterimages of the relevant objects throughout the surgical procedure, or atleast during navigation steps, if necessary. The tracking controller 60is therefore tasked with ensuring that the tracking device 40 is in asuitable position and orientation as a function of the surgicalworkflow, and controls an automated robotized variation of the positionand orientation of the tracking device 40 (e.g., by moving the trackingdevice 40, the bone, etc). For this purpose, the tracking controller 60may have different modules to assist the tracking calculator module 61in determining a desired position and orientation of the tracking device40, for the tracker arm 30 to be driven by the robot driver module 51into reaching that desired position and orientation.

According to one embodiment, the tracking controller 60 has a positionoptimizer module 62. The position optimizer module 62 may identifysituations when the backscatter images captured by the tracking device40 feature excessive environment in contrast to the objects to betracked (e.g., bone(s) and tool(s)), i.e., the ratio of tracked objectfor environment is not sufficiently high. For example, if the proportionof pixels in the backscatter images identified as being bone or tool isbelow a given threshold, the position optimizer module 62 may indicateto the tracking calculator module 61 that the tracking device 40 must berealigned, or recentered. As the position optimizer module 62 performsthe image analysis to identify the target zones in the backscatterimages, it may suggest suitable position and orientation for thetracking device 40 to increase the proportion of the tracked objects inthe images. The position optimizer module 62 may for example isolatesoft tissue from bone matter in the backscatter images. In someembodiments, the position optimizer module 62 discriminates between softtissue and bone matter in the backscatter images based on the spectralband of the backscattered light. For instance, light backscattered in afirst spectral band can be associated to soft tissue whereas lightbackscattered in a second spectral band can be associated to bonematter. Accordingly, the position optimizer module 62 can suggestsuitable position and orientation for the tracking device 40 to increasethe proportion of backscattered light in one of the first and secondspectral bands, depending on which one of the soft tissue and the bonematter is tracked. The action of the position optimizer module 62 maycause a dynamic adjustment of the position and orientation for thetracking device 40 during surgery.

The tracking controller 60 may have an interference identifier module63. The interference identifier module 63 may detect when interferenceoccurs in the line of sight between the tracking device 40 and thetracked objects. The interference may be of temporary nature, such asthe presence of an interfering object in the line of sight, or may be ofpermanent nature, such as soft tissue on the bone (e.g., cartilage, notpart of the virtual bone model B). The interference identifier module 63may determine the nature of the interference. For example, theinterference identifier module 63 may detect the appearance of an objectfrom a continuous tracking of the 3D geometry by the tracking calculatormodule 61. The interference identifier module 63 may also detect adiscrepancy between the virtual bone model B and the 3D geometry. If thediscrepancy has backscattering properties different than those of thesurrounding surfaces, the interference identifier module 63 may identifythe nature of the interference, such as cartilage or bodily fluids. Forinstance, the backscattering properties of the discrepancy may belong toa given spectral band which is known to be associated with soft tissuerather than bone matter. As a result of the identification ofinterference by the interference identifier module 63, the trackingcontroller 60 may ignore some types of interferences to proceed with thecontinuous tracking, may suggest a repositioning of the tracking device40 to an interference-less position and orientation or a position andorientation with a reduction thereof, and/or signal an interference tothe operator of the CAS system 10 via the interface D.

The tracking controller 60 may also have a field-of-view (FOV) navigatormodule 64. The FOV navigator module 64 may perform with the trackingcalculator module 61 a global field of view scan of the surgical site inthe early stages of the surgical workflow, and store same, forsubsequent reference by the tracking controller 60. This global FOV scanmay be particularly useful when the bones are fixed (e.g., with the footsupport A1 and the thigh support A2). The FOV navigator module 64 maytherefore store a correlation between the location of the trackingdevice 40 and the location of the objects in the coordinate system. As afunction of the surgical planning 52, the tracking controller 60 maydetermine a suitable position and orientation for the tracking device 40in anticipation of interventions of tools on the bone. The trackingcontroller 60 may know that the tool will be oriented and positioned ina given manner in the coordinate system relative to the bone accordingto the next upcoming step of the surgical planning 52, and may rely onthe FOV navigator module 64 to suggest a suitable position andorientation based on the FOV scan data.

Therefore, the tracking controller 60 continuously updates the positionand/or orientation of the patient bones and tools in the coordinatesystem using the data from the tracking device 40, and may ensure thatthe tracking is continuously updated by acting concurrently with therobot driver module 51 to adjust the position and/or orientation of thetracking device 40. Moreover, once alterations are done, the trackingperformed by the tracking controller 60 may be used to validate bonealterations, such as cut planes. In such a case, the surgical planning52 provides the planned alterations in the model of the bone. Thestructured light technology can determine the location of a cut planerelative to a remainder of the bone, and thus the tracking controller 60may determine of the cut plane is located according to planning, or if adiscrepancy is present. The tracking controller 60 may perform otherfunctions as well, such as selectively dimming or shutting off lights inthe operating room if excessive glare interfering with the trackingdevice 40 is detected. The tracking controller 60 may hence beinterfaced to the lighting system of the operating room in anembodiment, for instance with appropriate wireless protocols. Thetracking controller 60 may also send instructions via the interface D torequest adjustments to the ambient lighting system.

The surgical planning 52 may incorporate a navigation file for robotizedsurgery to calibrate the robot arm 20 and the tracking device 40 on thetracker arm 30 prior to commencing surgery. For example, the calibrationsubfile may include the virtual bone model B of the patient, for surfacematching to be performed by a registration pointer of the robot arm 20,used for contacting the bone. The robot arm 30 would obtain a cloud ofbone landmarks of the exposed bones, to reproduce a 3D surface of thebone. The 3D surface would then be matched to the bone model B of thepatient, to set the 3D model in the X, Y, Z coordinate system, incombination with concurrent optical tracking as described above. Anoperator's assistance may be requested initially, for instance toidentify tracked landmarks and focus the tracking device 40. This may bepart of the calibration subfile. The calibration pattern E and thetracking patterns F may also be part of the calibration subfile, if thetracking device 40 is a non-structured light ranging camera.

Referring back to FIG. 1, the secondary tracking device 70 mayoptionally be used to supplement the tracking done by the trackingdevice 40. For example, the secondary tracking device 70 may assist inproviding additional accuracy in relating the position and orientationof the tool head 24 to that of the tracking device 40, in the X, Y, Zcoordinate system. According to an embodiment, the secondary trackingdevice 70 comprises a camera that optically sees and recognizesretro-reflective markers 71A, 71B, 71C and/or 71D, with 71B and 71Coptionally used to track the limbs in six DOFs, namely in position andorientation. The marker 71A is on the tool head 24 of the robot arm 20such that its tracking allows the controller 50 to calculate theposition and/or orientation of the tool head 24 and tool 26A thereon.Likewise, marker 71D is on the tracking device 40 at the end of thetracker arm 30 such that its tracking allows the controller 50 tocalculate the position and/or orientation of the tracking device 40.Markers 71B and 71C are fixed to the patient bones, such as the tibiafor marker 71B and the femur for marker 71C. As shown, the markers 71Band 71C attached to the patient need not be invasively anchored to thebone, as straps or like attachment means may provide sufficient graspingto prevent movement between the markers 71B and 71C and the bones, inspite of being attached to soft tissue. However, the references 71B and71C could also be secured directly to the bones.

The markers 71A-D can be provided in the form of retro-reflectivemarkers or in the form of active emitters. In both cases, the filter 46of the tracking device 40 is designed so as to let pass spectral bandsassociated with the light reflected or emitted by the markers 71A-D suchas to be detectable by the camera(s) 42, if it is intended for thetracking device 40 to use these markers 71A-D. However, it may bedesired to use filters to block light reflected by the markers 71A-D toavoid interference with the operation of the tracking device 40 and thuswith the backscatter and structured light, in an embodiment in which thetracking device 40 and secondary tracking device 70 are usedindependently from one another.

In the illustrated embodiment, the markers 71A-D are retro-reflectivemarkers. Accordingly, the light source 48 is provided to illuminate themarkers 71A-D during the surgery. The light source 48 is adapted to emitlight which will be reflected by the retro-reflective markers 71A-D. Forinstance, if the markers 71A-D are passively reflecting markers, thelight source 48 is chosen to exhibit a spectral profile to betransmitted through the filter 46. Alternatively, if the markers 71A-Dare fluorescent markers, the light source 48 is selected to have aspectral profile suitable for generating fluorescence from the markers71A-D, and the filter 46 includes a spectral pass band for transmittingthe emitted fluorescence. One example of such markers includes passiveinfrared (IR) markers which are specifically designed to reflect lightin the infrared portion of the electromagnetic spectrum, in which casethe light source 48 is an IR light source. In the embodiment illustratedin FIG. 1, the light source 48 is made integral to the tracking device40. However, in other embodiments, the light source 48 can be separatefrom the tracking device 40.

As an alternative to optical tracking, the secondary tracking system 70may consist of inertial sensors (e.g., accelerometers, gyroscopes, etc)that produce tracking data to be used by the tracking controller 60 toassist in continuously updating the position and/or orientation of therobot arm 20. Other types of tracking technology may also be used. Whilethe secondary tracking system 70 may be present to assist in ensuringthe accuracy of the CAS system 10, the system 10 may also rely solely onthe combination of the tracking device 40 and the sensors on the robotarm 20 and the tracker arm 30 (e.g., encoders, etc) throughout thesurgical procedure. The combination of the tracking device 40 and thesensors on the robot arm 20 and the tracker arm 30 may provide redundanttracking data ensuring that the surgical procedure meets the requiredprecision and accuracy.

Referring to FIG. 3, a method for tracking one or more bones and objectssuch as tools, in robotized computer-assisted surgery, is generallyshown at 80. The method may be performed for instance by one or moreprocessors related to the CAS controller 50 and/or the trackingcontroller 60 (which may also be referred to as system), and operatingjointly with the robot driver 51. The method may be inscribed on anon-transitory computer-readable memory communicatively coupled to theprocessing unit of the CAS controller 50 and/or the tracking controller60, for example in the form of computer-readable program instructionsexecutable by the processing unit. According to 81, backscatter imagesof one or more bones are obtained, from a tracking device such as thetracking device 40, in the coordinate system. This may include obtainingimages of objects other than the bone, such as surgical tools. It mayalso include projecting structured light patterns on the objects totrack. The backscatter images may be used to calibrate the rangingcamera 42, for non-structured light ranging, along with the calibrationpattern E.

According to 82, a three-dimensional geometry of a surface of thebone(s) is generated from the backscatter images, the three-dimensionalgeometry of the surface being in the coordinate system. This may includegenerating a three-dimensional geometry of a surface of the surgicaltool from the backscatter images in the coordinate system.

According to 83, a position and orientation of the bone(s) is determinedin the coordinate system by matching the three-dimensional geometry ofthe surface of the at least one bone to a three-dimensional model of thebone. The position and orientation of the surgical tool may also bedetermined in the coordinate system by matching the three-dimensionalgeometry of the surface of the surgical tool to a three-dimensionalmodel of the surgical tool. A position and orientation of trackingpatterns F on the bone and/or tool may also be determined in thecoordinate system for subsequent dynamic tracking.

According to 84, an automated robotized variation of the position and/ororientation of the tracking device 40 is controlled, as a function of aprocessing of the backscatter images, for example to ensure continuousfield of view or to improve the tracking resolution. The controlling ofthe automated robotized variation may comprise identifying an imageratio of the bone in the backscatter images and controlling theautomated robotized variation to increase the image ratio. Thecontrolling of the automated robotized variation may comprises creatingfield of view data indicative of the orientation of the bone(s) as afunction of the position and orientation of the tracking device 40, andselecting the position and orientation of the tracking device 40 as afunction of a desired point of view, i.e., relative orientation, of thebone by the tracking device 40. The selection of the position andorientation of the tracking device 40 as a function of a desired pointof view of the at least bone may include determining from surgeryplanning a location of a robotized tool altering the bone.

According to 85, the position and orientation of the bone(s) in thecoordinate system is continuously output to the robot driver 51controlling the robot arm 20 supporting the surgical tool 24 in thecoordinate system for altering the bone. The position and orientation ofthe bone(s) in the coordinate system may be continuously output with theposition and orientation of the surgical tool in the coordinate system.The continuous output may include imaging of bone alterations, such ascut planes, for such bone alterations to be validated in comparison tosurgical planning 52.

Referring to FIG. 4, the non-structured light 3D camera 42 is shown, asbeing mounted directly to a tool, in this case to one of the retractors90. The retractors 90 are in close proximity to the operation site, andare generally immovable during the surgical procedure, whereby they forman efficient base for receiving the camera 42. This arrangement may alsobe used with the structured light head described above. It is hencecontemplated to mount the camera 42 directly to some tools. Forinstance, the camera 42 may be mounted directly to a drill head, etc, asit may often have a direct line of sight between the tool and the targetsite on the bone surface. Other examples include an endoscope,registration pointer, cutting tools, reamers, etc. The tracking for thearrangement of FIG. 4 may depend on the context. If the camera 42 has adirect and non-obstructed line of sight with the tool and target site,its position and orientation is not relevant. This applies to roboticand non-robotic applications. If it is mounted to a robotic arm, as inFIG. 1, the various tracking systems described above may be used. It isalso contemplated to provide an optic fiber with Braggs network 91 todetermine the position and orientation of the camera 42. It is alsocontemplated to provide tracking patterns F on implants as well, forinstance as integrated thereon, for precise validation of implantplacement.

While the description refers to the robot arm 20 as having a tool head24 and the tracker arm 30 as having the tracking device 40, it may bepossible to swap the tool head 24 and the tracking device 40. This maybe done for optimal placement of the tool head 24. For example, sometypes of procedures may benefit from such a swap, such as a bilateraltotal knee arthroplasty when the operation moves from one leg to thenext. The present disclosure refers to the system 10 has performingcontinuous tracking. This means that the tracking may be performedcontinuously during discrete time periods of a surgical procedure.Continuous tracking may entail pauses, for example when the bone is notbeing altered. However, when tracking is required, the system 10 mayprovide a continuous tracking output, with any disruption in thetracking output triggering an alarm or message to an operator.

The invention claimed is:
 1. A system for tracking at least one bone inrobotized computer-assisted surgery, comprising: a tracking device on arobotized tracker arm; a robot arm supporting a surgical tool; aprocessing unit; and a non-transitory computer-readable memorycommunicatively coupled to the processing unit and comprisingcomputer-readable program instructions executable by the processing unitfor: obtaining backscatter images of the at least one bone from atracking device in a coordinate system; generating a three-dimensionalgeometry of a surface of the at least one bone from the backscatterimages, the three-dimensional geometry of the surface being in thecoordinate system; determining a position and orientation of the atleast one bone in the coordinate system by matching thethree-dimensional geometry of the surface of the at least one bone to athree-dimensional model of the bone; controlling an automated robotizedvariation of at least one of a position and orientation of the trackingdevice on the robotized tracker arm as a function of a processing of thebackscatter images; and continuously outputting the position andorientation of the at least one bone in the coordinate system to a robotdriver controlling the robot arm supporting the surgical tool in thecoordinate system for altering the bone.
 2. The method according toclaim 1, wherein controlling the automated robotized variation comprisesidentifying an image ratio of the at least one bone relative toenvironment in the backscatter images and controlling the automatedrobotized variation to increase the image ratio.
 3. The system accordingto claim 1, wherein controlling the automated robotized variationcomprises creating field of view data indicative of the orientation ofthe at least one bone relative to the position and orientation of thetracking device, and selecting the position and orientation of thetracking device as a function of a desired point of view of the at leastbone.
 4. The system according to claim 3, wherein selecting the positionand orientation of the tracking device as a function of a desired pointof view of the at least bone includes determining an upcoming locationof the surgical tool on the robot arm from a surgical flow of surgeryplanning.
 5. The system according to claim 1, wherein: obtainingbackscatter images of the at least one bone from the tracking device inthe coordinate system comprises obtaining backscatter images of thesurgical tool; and generating the three-dimensional geometry of thesurface of the at least one bone from the backscatter images comprisesgenerating a three-dimensional geometry of a surface of the surgicaltool from the backscatter images in the coordinate system.
 6. The systemaccording claim 5, wherein continuously outputting the position andorientation of the at least one bone in the coordinate system includescontinuously outputting the position and orientation of the surgicaltool to the robot driver controlling the robot arm supporting thesurgical tool in the coordinate system.
 7. The system according to claim6, further comprising verifying the position and orientation of thesurgical tool obtained from the backscatter images with a position andorientation of the surgical tool provided by the robot driver andoutputting a discrepancy.
 8. The system according to claim 5, whereindetermining the position and orientation of the at least one bone in thecoordinate system includes determining the position and orientation ofthe surgical tool in the coordinate system by matching thethree-dimensional geometry of the surface of the surgical tool to athree-dimensional model of the surgical tool.
 9. The system according toclaim 1, wherein obtaining backscatter images of the at least one bonefrom a tracking device in a coordinate system includes obtaining thebackscatter images in a first frequency capture mode when the surgicaltool is distal to the bone, and in a second frequency capture mode whenthe surgical tool is proximal to the bone, the second frequency capturemode being greater than the first frequency capture mode.
 10. The systemaccording to claim 9, wherein obtaining the backscatter images in thefirst frequency capture mode or the second frequency capture modeincludes determining an upcoming location of the surgical tool on therobot arm from a surgical flow of surgery planning.
 11. The systemaccording to claim 1, wherein obtaining the backscatter images of the atleast one bone includes obtaining the backscatter images from a point ofview on a tool interfacing with an anatomical part surrounding the boneor with the bone.
 12. The system according to claim 1, whereincontrolling the automated robotized variation of at least one of theposition and orientation of the tracking device includes controlling therobotized tracker arm of the tracking device.
 13. The system accordingto claim 1, wherein generating the three-dimensional geometry of thesurface of the at least one bone from the backscatter images includesgenerating the three-dimensional geometry of an altered surface, andwherein determining the position and orientation of the at least onebone in the coordinate system includes determining the position andorientation of the altered surface in the bone.
 14. The system accordingto claim 13, further comprising verifying the position and orientationof altered surface with a position and orientation of a planned alteredsurface from surgery planning and outputting a discrepancy.
 15. Thesystem according to claim 1, further comprising identifying aninterference from the backscatter images of the at least one bone andoutputting an indication of interference.
 16. The system according toclaim 1, wherein obtaining the backscatter images includes obtaining thebackscatter images from visible structured light.
 17. A system fortracking at least one bone in robotized computer-assisted surgery,comprising: a tracking device including at least a structured lightsource and a camera to generate and obtain the backscatter images, thetracking device being on the robotized tracker arm a robot armsupporting a surgical tool; and a computer-assisted surgery controllerfor obtaining backscatter images of the at least one bone from thetracking device in a coordinate system, generating a three-dimensionalgeometry of a surface of the at least one bone from the backscatterimages, the three-dimensional geometry of the surface being in thecoordinate system, determining a position and orientation of the atleast one bone in the coordinate system by matching thethree-dimensional geometry of the surface of the at least one bone to athree-dimensional model of the bone, controlling an automated robotizedvariation of at least one of a position and orientation of the trackingdevice as a function of a processing of the backscatter images, andcontinuously outputting the position and orientation of the at least onebone in the coordinate system to a robot driver controlling the robotarm supporting Hall the surgical tool in the coordinate system foraltering the bone.
 18. A method for tracking at least one bone incomputer-assisted surgery, comprising: obtaining backscatter images ofthe at least one bone from a tracking device in a coordinate system, aspositioned on a tool interfacing with an anatomical part surrounding thebone or with the bone; generating a three-dimensional geometry of asurface of the at least one bone from the backscatter images, thethree-dimensional geometry of the surface being in the coordinatesystem; determining a position and orientation of the at least one bonein the coordinate system by matching the three-dimensional geometry ofthe surface of the at least one bone to a three-dimensional model of thebone; and continuously outputting the position and orientation of the atleast one bone in the coordinate system to an interface for providingguidance in altering the bone.